Nonparametric Compositional Stochastic Optimization for Risk-Sensitive Kernel Learning

In this work, we address optimization problems where the objective function is a nonlinear of an expected value, i.e., compositional stochastic programs. We consider case decision variable not vector-valued but instead belongs to Reproducing Kernel Hilbert Space (RKHS), motivated by risk-aware formulations supervised learning. develop first memory-efficient algorithm for setting, which call Compositional Online Learning with Kernels (COLK). COLK, at its core two time-scale approximation method, addresses facts that (i) compositions value cannot be addressed gradient method due presence inner expectation; and (ii) RKHS-induced parameterization has complexity proportional iteration index mitigated through greedily constructed subspace projections. provide, time, non-asymptotic tradeoff between required convergence accuracy both strongly convex non-convex objectives under constant step-sizes. Experiments risk-sensitive learning demonstrate COLK consistently converges performs reliably even when data full outliers, thus marks step towards overfitting. Specifically, observe favorable model complexity, consistent convergence, statistical associated heavy-tailed distributions.